Metrics for Explainable AI: Challenges and Prospects (1812.04608v2)

Abstract: The question addressed in this paper is: If we present to a user an AI system that explains how it works, how do we know whether the explanation works and the user has achieved a pragmatic understanding of the AI? In other words, how do we know that an explanainable AI system (XAI) is any good? Our focus is on the key concepts of measurement. We discuss specific methods for evaluating: (1) the goodness of explanations, (2) whether users are satisfied by explanations, (3) how well users understand the AI systems, (4) how curiosity motivates the search for explanations, (5) whether the user's trust and reliance on the AI are appropriate, and finally, (6) how the human-XAI work system performs. The recommendations we present derive from our integration of extensive research literatures and our own psychometric evaluations.

Metrics for Explainable AI: Challenges and Prospects

This paper discusses the critical issue of evaluating Explainable AI (XAI) systems, particularly in determining whether an AI-generated explanation effectively communicates to the user. The authors, Hoffman, Mueller, Klein, and Litman, delve into key concepts of measurement pertaining to the evaluation of explanation quality, user satisfaction, understanding, curiosity, trust, and human-AI system performance. Their work is informed by extensive literature reviews and their own psychometric evaluations.

Key Themes

1. Explanation Goodness and Satisfaction
   • The paper differentiates between explanation goodness (an a priori evaluation by researchers) and explanation satisfaction (a posteriori evaluation by users). An Explanation Goodness Checklist and an Explanation Satisfaction Scale are proposed to assess these factors. The distinction underscores the importance of context in evaluating whether an explanation is useful to a user.
2. Mental Models
   • The authors emphasize the importance of users forming mental models to understand AI systems. Various methods for eliciting these mental models are discussed, emphasizing their role in fostering appropriate trust and improving user performance with AI systems.
3. Curiosity and Exploration
   • Curiosity drives the demand for explanations, and the paper discusses how explanations should harness curiosity to close knowledge gaps. The potential for explanations to either stimulate or suppress curiosity is critically analyzed, with tools like the Curiosity Checklist introduced to measure users' intrinsic motivations for seeking explanations.
4. Trust and Reliance
   • Trust is multifaceted, encompassing factors like reliability, predictability, safety, and efficiency. The authors propose a trust measurement scale tailored for the XAI context. Trust metrics are designed to capture both positive and negative trusting states, thereby supporting the calibration of appropriate reliance on AI systems.
5. Performance Evaluation
   • The paper argues that evaluating user performance with AI is inextricably linked to assessing explanations and mental models. Performance is measured through task success rates and alignment with primary task goals, considering both user actions and joint human-machine system effectiveness.

Methodological Insights

The authors adopt a multi-method approach, combining quantitative and qualitative assessments to understand XAI effectiveness comprehensively. Psychometric validation of scales ensures reliable and meaningful measures of explanation and trust.

Implications and Future Perspectives

The findings highlight the complexity of developing XAI systems that are not only capable of providing technically correct explanations but also psychologically satisfying and contextually useful ones. The multi-dimensionality of trust, alongside curiosity-driven engagement with AI explanations, points to a need for adaptive and user-centered XAI systems. Future research could further explore the dynamics between trust, reliance, and explanation clarity in real-world applications.

Conclusion

Overall, the paper provides a foundational framework for developing metrics for XAI that are both sophisticated and grounded in empirical research. By integrating insights across cognitive psychology, philosophy, and human-computer interaction, the authors pave the way for nuanced evaluations of AI explainability, propelling both theoretical advancements and practical applications in the field of AI.

In summary, this paper offers a thorough and methodical examination of the complexities involved in measuring the quality and efficacy of explanations provided by AI systems, setting the stage for further academic inquiry and practical innovations in XAI.