Explainable Reinforcement Learning Through a Causal Lens (1905.10958v2)

Abstract: Prevalent theories in cognitive science propose that humans understand and represent the knowledge of the world through causal relationships. In making sense of the world, we build causal models in our mind to encode cause-effect relations of events and use these to explain why new events happen. In this paper, we use causal models to derive causal explanations of behaviour of reinforcement learning agents. We present an approach that learns a structural causal model during reinforcement learning and encodes causal relationships between variables of interest. This model is then used to generate explanations of behaviour based on counterfactual analysis of the causal model. We report on a study with 120 participants who observe agents playing a real-time strategy game (Starcraft II) and then receive explanations of the agents' behaviour. We investigated: 1) participants' understanding gained by explanations through task prediction; 2) explanation satisfaction and 3) trust. Our results show that causal model explanations perform better on these measures compared to two other baseline explanation models.

An Overview of "Explainable Reinforcement Learning Through a Causal Lens"

The paper "Explainable Reinforcement Learning Through a Causal Lens" by Prashan Madumal et al. presents a novel approach to generating explanations for the behavior of model-free reinforcement learning (RL) agents using causal models. The authors propose an action influence model that integrates structural causal models (SCMs) to derive causal explanations from the behavior of RL agents. This approach aims to improve the transparency and interpretability of AI systems, grounded in cognitive science theories that link human explanations to causal reasoning.

Key Contributions

1. Action Influence Model: The paper introduces an action influence model for RL agents based on structural causal models. This model encodes causal relationships between variables relevant to the agent’s environment, allowing for the generation of explanations that answer "why" and "why not" questions. These explanations are based on the causal structure of the agent's decision-making processes, enabling a deeper understanding of the actions taken by the agent.
2. Minimally Complete Explanations: The authors define minimally complete explanations that balance completeness and brevity, focusing on the essential causal factors driving an agent's decisions. This approach is inspired by social psychology and aims to avoid overwhelming users with unnecessary details.
3. Algorithm and Evaluation: An algorithm to generate explanations from causal models is presented, where structural equations are learned during the RL process. The authors conducted computational evaluations in six RL benchmarks, demonstrating reasonable accuracy in task prediction without significant performance degradation. A human paper with 120 participants was also conducted, showing that causal model explanations lead to better understanding and task prediction abilities compared to baseline models.

Numerical Results and Claims

The paper reports strong numerical results in both computational evaluations and human studies. In computational evaluations, the proposed model achieves high task prediction accuracy, with minimal performance impact, particularly in environments with clear causal structure like Starcraft II. In human studies, the causal explanations lead to statistically significant improvements in participants' ability to predict the agent's future actions when compared to baseline explanation models. However, the authors did not find significant differences in trust levels, suggesting a complex relationship between understanding and trust.

Implications and Future Directions

The implications of the paper’s findings are substantial for the field of Explainable AI (XAI). The approach of using causal models for generating explanations aligns with the human cognitive process of understanding through causality and counterfactuals. This alignment suggests that these models could significantly enhance user satisfaction with AI explanations, contributing to better human-AI collaboration.

Future research could explore several directions, including:

• Extending the proposed model to handle continuous domains and actions.
• Incorporating explainees’ prior knowledge to tailor explanations based on their epistemic state.
• Investigating the integration of causal explanation models with other XAI techniques to enhance both interpretability and trust further.

Enabling AI systems to provide causal explanations is not only essential for gaining user trust but also for adhering to ethical AI guidelines that emphasize transparency and accountability. The robust framework established by this paper offers a promising pathway for future research in explainable reinforcement learning.