The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies (2007.15911v2)

Abstract: AI has huge potential to improve the health and well-being of people, but adoption in clinical practice is still limited. Lack of transparency is identified as one of the main barriers to implementation, as clinicians should be confident the AI system can be trusted. Explainable AI has the potential to overcome this issue and can be a step towards trustworthy AI. In this paper we review the recent literature to provide guidance to researchers and practitioners on the design of explainable AI systems for the health-care domain and contribute to formalization of the field of explainable AI. We argue the reason to demand explainability determines what should be explained as this determines the relative importance of the properties of explainability (i.e. interpretability and fidelity). Based on this, we propose a framework to guide the choice between classes of explainable AI methods (explainable modelling versus post-hoc explanation; model-based, attribution-based, or example-based explanations; global and local explanations). Furthermore, we find that quantitative evaluation metrics, which are important for objective standardized evaluation, are still lacking for some properties (e.g. clarity) and types of explanations (e.g. example-based methods). We conclude that explainable modelling can contribute to trustworthy AI, but the benefits of explainability still need to be proven in practice and complementary measures might be needed to create trustworthy AI in health care (e.g. reporting data quality, performing extensive (external) validation, and regulation).

The Role of Explainability in Trustworthy AI for Health Care: A Detailed Survey

In the intricate landscape of health care AI, the potential for AI applications to significantly improve patient outcomes is immense. Yet, one of the key hurdles limiting its widespread adoption is the trust deficit due to the opacity of AI systems. The paper "The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies" by Markus et al. explores this issue by examining how explainability can bridge this trust gap and aid in designing reliable AI systems.

The authors provide a rigorous survey of explainable AI (XAI) methodologies tailored to the health-care domain. The core tenet is that the necessity for explainability determines the aspects of AI that need elucidation, balancing interpretability and fidelity. An innovative framework is proposed, offering researchers a decision pathway between different classes of XAI, namely explainable modeling and post-hoc explanations, along with subdivisions such as model-based, attribution-based, and example-based approaches.

Explainability and its Impact on Trustworthiness

The paper highlights the pressing need for explainability as a means to foster trust in AI, particularly in high-stakes environments like health care. By unraveling the often opaque decision-making processes of AI models, clinicians can utilize AI-driven insights with higher assurance. However, the authors caution that the benefits of explainability are yet to be conclusively validated in practical scenarios. They posit that explainable modeling, which inherently interprets model decisions, may be more suited to contexts requiring high fidelity and interpretability. This contrasts with post-hoc methods, which, while potentially more powerful in predictive accuracy, might present misleading explanations if not properly validated.

Evaluation and Selection of Explainable AI Techniques

A critical analysis within the paper identifies a gap in standardized evaluation metrics for some explanation properties like clarity and example-based methods. The authors stress the necessity for quantitative metrics, which are essential for an objective evaluation of explainability and comparing methodologies.

The authors' framework for selecting explainable AI methods involves a series of decision steps. Starting from the relative importance of explainability vis-a-vis predictive performance, the decision path directs developers through choices that ensure high fidelity and robust interpretability. The framework underscores the imperative to align the chosen methods with the specific needs of the clinical domain, emphasizing that transparency should not compromise accurate AI outcomes.

Implications and Future Directions

The implications of this comprehensive survey are manifold. Practically, the paper provides actionable insights into design principles that could significantly enhance the adoption of AI in health care by improving trust through explainability. Theoretically, it stresses the synergy required between explainable models and complementary measures—such as rigorous validation and data quality assessment—to establish truly trustworthy AI systems. Moreover, it suggests future research should focus on refining explainable modeling methods and further exploring regulative measures specific to AI's evolving nature.

The findings of Markus et al. continue to remain relevant as the field progresses. Explainable AI represents an essential yet complex aspect of AI development, pivotal for its ethical and effective integration into health care. This paper serves as a cornerstone for ongoing discussions and research efforts aiming to reconcile high-performance AI with the critical requirement for transparency and accountability in decision-making systems.