What do we need to build explainable AI systems for the medical domain? (1712.09923v1)

Abstract: AI generally and ML specifically demonstrate impressive practical success in many different application domains, e.g. in autonomous driving, speech recognition, or recommender systems. Deep learning approaches, trained on extremely large data sets or using reinforcement learning methods have even exceeded human performance in visual tasks, particularly on playing games such as Atari, or mastering the game of Go. Even in the medical domain there are remarkable results. The central problem of such models is that they are regarded as black-box models and even if we understand the underlying mathematical principles, they lack an explicit declarative knowledge representation, hence have difficulty in generating the underlying explanatory structures. This calls for systems enabling to make decisions transparent, understandable and explainable. A huge motivation for our approach are rising legal and privacy aspects. The new European General Data Protection Regulation entering into force on May 25th 2018, will make black-box approaches difficult to use in business. This does not imply a ban on automatic learning approaches or an obligation to explain everything all the time, however, there must be a possibility to make the results re-traceable on demand. In this paper we outline some of our research topics in the context of the relatively new area of explainable-AI with a focus on the application in medicine, which is a very special domain. This is due to the fact that medical professionals are working mostly with distributed heterogeneous and complex sources of data. In this paper we concentrate on three sources: images, *omics data and text. We argue that research in explainable-AI would generally help to facilitate the implementation of AI/ML in the medical domain, and specifically help to facilitate transparency and trust.

Explainable AI for the Medical Domain

The paper "What do we need to build explainable AI systems for the medical domain?" by Holzinger et al. provides a comprehensive exploration of the requirements for constructing explainable AI (XAI) systems with a focus on applications within medicine. The work addresses a critical issue: as AI and ML technologies become increasingly integral to medical tasks, their inherent "black-box" nature poses significant challenges for interpretability and trust. This paper endeavors to outline essential considerations for developing AI systems that produce understandable and transparent results.

Key Insights and Findings

The authors identify the tension between algorithmic performance and explainability. High-performance models like deep learning architectures often lack transparency, impeding user trust, especially crucial in medical settings where decision stakes are high. The research underscores the importance of making AI outcomes retraceable to foster trust among medical professionals.

Particular attention is given to the need to integrate and interpret diverse data types—images, *omics data, and text—prevalent in medical environments. By enabling explainability, medical professionals can better understand AI-driven decisions, enhancing their ability to integrate AI insights into clinical workflows and decision-making processes.

Explainability Techniques

The paper classifies explainability into post-hoc and ante-hoc methods:

1. Post-hoc Explainability: Techniques like LIME and BETA are highlighted. These models explain individual predictions by providing local approximations that are human-interpretable without altering the original model's overall structure.
2. Ante-hoc Explainability: Ante-hoc methods, such as generalized additive models and fuzzy systems, integrate explainability into their structure from inception, providing inherently interpretable models.

The authors illustrate these approaches using a variety of case studies, including deep neural networks and their application in scenarios requiring visualization of intermediate steps.

Applications and Implications

The paper discusses the implementation of AM-FM decompositions for medical image analysis, offering a method to obtain meaningful representations of complex medical images—such techniques are vital for understanding and communicating findings in contexts like digital pathology.

Furthermore, the integration of *omics data enriches the analytical capabilities of AI models, facilitating the investigation of complex biological mechanisms through a spectrum of genomic, proteomic, and metabolomic data. The authors emphasize the potential of combining dense neural representations with sparse graphical models to benefit from both efficiency and interpretability.

Future Directions

Looking forward, the paper suggests expanding hybrid approaches that blend rule-based logic with deep learning to enhance both performance and explainability. The authors propose ongoing collaborations with medical professionals to refine AI systems that augment human expertise without bypassing it.

The integration of human-in-the-loop strategies emerges as a promising avenue, suggesting that interactive systems can adaptively learn from human guidance, thereby aiding in constructing more reliable models tailored to specific medical domains.

Conclusion

This paper concludes that while the path to fully explainable AI systems in medicine is complex, strides towards integrating transparent, reliable, and interpretable models are both necessary and underway. Legal and ethical considerations, especially with evolving data protection regulations, amplify the urgency for these developments. The authors position explainable AI not just as a technical challenge but a multidisciplinary endeavor requiring collaboration across fields to build systems that are not only efficient but also trustworthy and aligned with human values in healthcare.