Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions (1901.03729v1)

Abstract: Automated rationale generation is an approach for real-time explanation generation whereby a computational model learns to translate an autonomous agent's internal state and action data representations into natural language. Training on human explanation data can enable agents to learn to generate human-like explanations for their behavior. In this paper, using the context of an agent that plays Frogger, we describe (a) how to collect a corpus of explanations, (b) how to train a neural rationale generator to produce different styles of rationales, and (c) how people perceive these rationales. We conducted two user studies. The first study establishes the plausibility of each type of generated rationale and situates their user perceptions along the dimensions of confidence, humanlike-ness, adequate justification, and understandability. The second study further explores user preferences between the generated rationales with regard to confidence in the autonomous agent, communicating failure and unexpected behavior. Overall, we find alignment between the intended differences in features of the generated rationales and the perceived differences by users. Moreover, context permitting, participants preferred detailed rationales to form a stable mental model of the agent's behavior.

Insights into Automated Rationale Generation for Explainable AI

The paper "Automated Rationale Generation: A Technique for Explainable AI and its Effects on Human Perceptions" presents an approach to creating human-like explanations for actions taken by autonomous agents in sequential environments. This technique, dubbed Automated Rationale Generation (ARG), seeks to bridge the gap between an agent's decision-making process and human users' understanding by translating internal state and action data into natural language explanations.

Context and Motivation

Explainable AI (XAI) is pivotal in enhancing trust and collaboration between humans and autonomous systems. As many autonomous systems operate in sequential environments where past decisions influence future actions, there is a pressing need to generate explanations that consider these temporal dependencies. The ARG technique is proposed as a method to address this challenge by developing rationales that resonate with human understanding, rather than mere technical elucidations of the agent's internal workings.

Methodology

This paper utilizes an agent trained to play the game Frogger, serving as a testbed for deploying and studying rationale generation. The process involves three primary components:

1. Data Collection and Corpus Building: The authors devised a robust interface to collect a corpus of human-generated explanations through think-aloud protocols. Participants played Frogger and provided natural language rationales for their actions, which were automatically transcribed and linked with game states.
2. Neural Translation Model: Using this corpus, the authors developed an encoder-decoder neural network to translate state-action representations into natural language rationales. Two configurations were explored: the focused-view, which emphasizes local context, and the complete-view, which considers broader game contexts.
3. User Study for Perception Analysis: Two user studies were conducted. Initially, a comparison was made between the generated rationales and randomly selected rationales, with results indicating a marked preference for the generated rationales. The second paper directly compared the focused-view and complete-view rationales, analyzing user perceptions along dimensions such as confidence, awareness, and strategic detail.

Results and Discussion

The findings demonstrate that rationale generation significantly affects user perceptions in dimensions critical to trust in autonomous agents. Participants generally favored detailed and holistic rationales, as these promoted a greater understanding and confidence in the agent, especially in scenarios involving failure or unexpected behavior. This indicates that users prefer rationales that offer a comprehensive view of the agent's decision-making process rather than those restricted to immediate contextual information.

Moreover, the studies validated the distinctions in design intent between the two configurations: the focused-view rationales were perceived as concise and localized, while the complete-view rationales were seen as detailed and holistic. These distinctions align with the underlying neural configuration strategies.

Implications and Future Directions

The research highlights essential insights for the design of explainable systems, suggesting that tailoring explanations to include comprehensive contextual details can improve user trust and interaction with AI systems. This holds significant implications for the deployment of AI in domains where understanding the reasoning behind decisions is critical, such as autonomous driving and healthcare.

Future work should address the limitations outlined, such as exploring interactivity in explanations and extending the methodology to more complex environments. Additionally, a longitudinal paper to assess the sustained impact of rationale generation on user trust could provide deeper insights.

In conclusion, this paper makes a substantial contribution by introducing and evaluating a technique for generating human-like rationales in sequential environments. Their approach indicates a promising direction for enhancing the interpretability and acceptance of AI systems through contextually rich explanations.