Machine Explanations and Human Understanding (2202.04092v3)

Abstract: Explanations are hypothesized to improve human understanding of machine learning models and achieve a variety of desirable outcomes, ranging from model debugging to enhancing human decision making. However, empirical studies have found mixed and even negative results. An open question, therefore, is under what conditions explanations can improve human understanding and in what way. Using adapted causal diagrams, we provide a formal characterization of the interplay between machine explanations and human understanding, and show how human intuitions play a central role in enabling human understanding. Specifically, we identify three core concepts of interest that cover all existing quantitative measures of understanding in the context of human-AI decision making: task decision boundary, model decision boundary, and model error. Our key result is that without assumptions about task-specific intuitions, explanations may potentially improve human understanding of model decision boundary, but they cannot improve human understanding of task decision boundary or model error. To achieve complementary human-AI performance, we articulate possible ways on how explanations need to work with human intuitions. For instance, human intuitions about the relevance of features (e.g., education is more important than age in predicting a person's income) can be critical in detecting model error. We validate the importance of human intuitions in shaping the outcome of machine explanations with empirical human-subject studies. Overall, our work provides a general framework along with actionable implications for future algorithmic development and empirical experiments of machine explanations.

Analysis of "Machine Explanations and Human Understanding"

The paper under discussion, titled "Machine Explanations and Human Understanding," provides a meticulous exploration of the conditions under which explanations from ML models enhance human understanding. The research aims to clarify the effects of machine explanations on human cognition, addressing the ambiguous results previously reported in empirical studies. Through rigorous theoretical modeling and validation via human-subject studies, the authors propose a new framework central to the intersection between human intuition and machine explanations.

Core Concepts and Theoretical Framework

The paper delineates three pivotal concepts for measuring human understanding in decision-making involving AI: the task decision boundary, the model decision boundary, and model error. These concepts form the foundation of a novel theoretical framework designed to dissect human understanding in the context of AI-related tasks. The authors adapt causal diagrams to formalize relationships among these concepts and their human approximations. This formalism allows for a structured examination of how different conditions impact human understanding.

The Role of Human Intuition

A crucial hypothesis presented is that human intuitions are indispensable for generating and evaluating machine explanations in human-AI decision making. The paper argues that without assumptions relating to human intuitions, explanations can potentially improve understanding of a model's decision boundary but fail to enhance human grasp of the task decision boundary or model error. This finding underscores a nuanced view that the utility of explanations is not universal but contingent upon specific cognitive preconditions.

Empirical Validation and Findings

To substantiate their theoretical claims, the authors conduct human-subject experiments, notably employing a Wizard-of-Oz setup to isolate and control variables regarding human intuition. The experimental data illustrate key results: in scenarios where human intuitions are diminished, individuals exhibit greater alignment with AI predictions, showcasing a dependence on machine explanations when intuitive guidance is absent. Conversely, when explanations align with human intuitions, there is heightened concordance between human and model judgments.

Implications for AI and Future Directions

The implications of this paper span both theoretical and practical domains. On a theoretical level, it contributes a refined paradigm for designing behavioral studies and evaluating machine explanations within AI-assisted decision making. Practically, the research suggests pathways for developing explanation systems that better leverage human intuition to improve decision outcomes. This introduces a critical awareness of incorporating human cognitive elements into the design of AI explanations.

Future research directions may delve into broader spectrum tasks beyond classification and engage with more intricate human cognitive models. The potential integration of probabilistic frameworks could offer further depth to understanding global human-model interactions. Moreover, developing adaptive systems capable of tailoring explanations based on user-specific intuitions and expertise levels represents an exciting frontier emerging from this work.

In sum, "Machine Explanations and Human Understanding" presents a sophisticated and thought-provoking exploration of cognitive interactions in human-AI settings. It forms a benchmark paper for researchers seeking to unravel the complexities of how machine-generated explanations align with and inform human understanding in the AI landscape.