Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Explainable Human-AI Interaction: A Planning Perspective (2405.15804v1)

Published 19 May 2024 in cs.AI

Abstract: From its inception, AI has had a rather ambivalent relationship with humans -- swinging between their augmentation and replacement. Now, as AI technologies enter our everyday lives at an ever increasing pace, there is a greater need for AI systems to work synergistically with humans. One critical requirement for such synergistic human-AI interaction is that the AI systems be explainable to the humans in the loop. To do this effectively, AI agents need to go beyond planning with their own models of the world, and take into account the mental model of the human in the loop. Drawing from several years of research in our lab, we will discuss how the AI agent can use these mental models to either conform to human expectations, or change those expectations through explanatory communication. While the main focus of the book is on cooperative scenarios, we will point out how the same mental models can be used for obfuscation and deception. Although the book is primarily driven by our own research in these areas, in every chapter, we will provide ample connections to relevant research from other groups.

PDF HTML Abstract

Explainable Human-AI Interaction: A Planning Perspective

The discussed paper, "Explainable Human-AI Interaction: A Planning Perspective" by Sarath Sreedharan, Anagha Kulkarni, and Subbarao Kambhampati, from Arizona State University, centers around the growing necessity for AI systems to interact transparently and cooperatively with humans. This necessity roots from the fact that as AI systems become increasingly pervasive in everyday applications, their decision-making processes must be comprehensible to human stakeholders. The paper explores the aspects and methodologies of creating AI systems that can explain their actions, decisions, and plans to humans effectively, emphasizing cooperative human-AI interactions, obfuscation, and deception scenarios.

Overview

The paper systematically explores the field of explainability in AI through various dimensions. It outlines that unlike conventional AI systems designed to function remotely or adversarially compared to humans (e.g., AI in games like Chess or Go), the goal is to develop systems that can actively engage and build trust with humans. This can be particularly critical in high-stakes domains such as healthcare or criminal justice where AI decisions significantly impact human lives.

Dimensions of Explainable AI Systems

The paper identifies multiple dimensions along which explainable AI systems can be evaluated:

  1. Explicability: The extent to which an AI’s actions align with human expectations.
  2. Legibility: The ability of an AI system to signal its goals or plans through its actions.
  3. Predictability: Ensuring that an AI's actions can be anticipated by human observers.

Each dimension highlights a different aspect of interaction between humans and AI, focusing on reducing the cognitive load for humans to understand AI behavior and improving the overall human-AI teaming efficiency.

Explanation Framework

A crucial aspect of the discussion focuses on model reconciliation. Instead of modifying the AI's plan to meet human expectations (as in explicable planning), the AI provides explanations to humans, thereby altering their understanding and expectations. This is achieved by communicating relevant aspects of the AI’s model that the human may not be aware of. The goal is to generate minimally complete explanations (MCE) that are concise but sufficient to make the given plans understandable and seem optimal to humans.

Approximate Explanations

The paper addresses how explanations and the necessary model information can be adjusted to account for the human observer's limited inferential capabilities. It discusses the trade-offs involved in finding the balance between the size of the explanation and the computational overhead involved in generating them.

Acquisition of Mental Models for Explanations

Detailed methods are proposed to handle scenarios where an AI does not possess an accurate model of the human's mental state initially. These methods include:

  1. Incomplete Models: Addressing situations where partial information about the human’s mental model is known.
  2. Model-Free Approaches: Learning approximate models from human feedback.
  3. Prototypical Models: Assuming simpler representations of human mental models for ease of explanation.
  4. Annotation and Robustness: Employing annotated models to gauge the robustness of explanations across possible human mental models.

Implications and Future Directions

The practical implications of this research are immense. In real-world scenarios such as urban search and rescue, medical decision support systems, and autonomous driving, the capability of an AI system to explain its actions fosters transparency and trust, crucial for human acceptance and collaboration.

Furthermore, the paper explores environmental design as a means to facilitate explicable behavior in repetitive tasks, emphasizing the importance of a synergistic relationship between environment modification and human-AI interaction strategies.

Theoretical and Practical Contributions

On a theoretical front, this research bridges the gap between AI planning and human-computer interaction (HCI), proposing comprehensive frameworks that formalize explainable behaviors and explanation generation mechanisms. On a practical level, the described methods and algorithms can be employed to develop more transparent and trustable AI systems.

Conclusion

The pursuit of explainable AI systems is critical for the integration of AI into domains where human collaboration is essential. This paper provides a detailed roadmap for achieving this through robust planning, explanation strategies, and mental model reconciliation, catering to both explanatory needs and computational efficiency. Future developments may focus on integrating more complex human cognitive models, allowing for even more nuanced and effective explanations.

By instilling the ability to explain, this research aims to foster AI systems that not only perform optimally but do so in a manner that engenders human trust and collaboration, marking significant strides towards human-aware AI systems.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (93)
  1. James F Allen. Mixed initiative planning: Position paper. In ARPA/Rome Labs Planning Initiative Workshop, 1994.
  2. Guidelines for human-ai interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019, page 3, 2019.
  3. Highlights: Summarizing agent behavior to people. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages 1168–1176, 2018.
  4. Modeling human plan recognition using bayesian theory of mind. Plan, activity, and intent recognition: Theory and practice, 7:177–204, 2014.
  5. Plan, Repair, Execute, Explain – How Planning Helps to Assemble Your Home Theater. In ICAPS, 2014.
  6. Belief tracking for planning with sensing: Width, complexity and approximations. Journal of Artificial Intelligence Research, 50:923–970, 2014.
  7. Cynthia Breazeal. Toward sociable robots. Robotics and autonomous systems, 42(3-4):167–175, 2003.
  8. Cynthia L Breazeal. Designing sociable robots. MIT press, 2004.
  9. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  10. Maintaining evolving domain models. In Proceedings of the twenty-fifth international joint conference on artificial intelligence, pages 3053–3059, 2016.
  11. (How) Can AI Bots Lie? In XAIP Workshop, 2019a.
  12. (when) can ai bots lie? In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 53–59, 2019b.
  13. Planning for serendipity. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5300–5306. IEEE, 2015.
  14. Plan Explanations as Model Reconciliation: Moving Beyond Explanation as Soliloquy. In IJCAI, 2017a.
  15. Mr. jones–towards a proactive smart room orchestrator. In AAAI Fall Symposia, 2017b.
  16. Projection-Aware Task Planning and Execution for Human-in-the-Loop Operation of Robots. In IROS, 2018.
  17. Explicability? Legibility? Predictability? Transparency? Privacy? Security?: The Emerging Landscape of Interpretable Agent Behavior. In ICAPS, 2019a.
  18. Plan explanations as model reconciliation–an empirical study. In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 258–266. IEEE, 2019b.
  19. Balancing Explicability and Explanation in Human-Aware Planning. In IJCAI, 2019c.
  20. The emerging landscape of explainable ai planning and decision making. In IJCAI, 2020.
  21. Generating Legible Motion. In RSS, 2013.
  22. Anca D Dragan. Robot Planning with Mathematical Models of Human State and Action. arXiv:1705.04226, 2017.
  23. Legibility and Predictability of Robot Motion. In HRI, 2013.
  24. Effects of Robot Motion on Human-Robot Collaboration. In HRI, 2015.
  25. A New Approach to Plan-Space Explanation: Analyzing Plan-Property Dependencies in Oversubscription Planning. In AAAI, 2020.
  26. Christiane Fellbaum. Wordnet. In Theory and applications of ontology: computer applications, pages 231–243. Springer, 2010.
  27. Generating Plans that Predict Themselves. In WAFR, 2018.
  28. Explainable Planning. In IJCAI XAI Workshop, 2017.
  29. A Concise Introduction to Models and Methods for Automated Planning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 2013.
  30. Deep learning. MIT press, 2016.
  31. Model elicitation through direct questioning. arXiv preprint arXiv:2011.12262, 2020.
  32. Explanation augmented feedback in human-in-the-loop reinforcement learning. arXiv preprint arXiv:2006.14804, 2020.
  33. Improving robot controller transparency through autonomous policy explanation. In 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 303–312. IEEE, 2017.
  34. Malte Helmert. The Fast Downward Planning System. JAIR, 2006.
  35. Jörg Hoffmann. Ff: The fast-forward planning system. AI magazine, 22(3):57–57, 2001.
  36. Jörg Hoffmann. Where’ignoring delete lists’ works: Local search topology in planning benchmarks. Journal of Artificial Intelligence Research, 24:685–758, 2005.
  37. Oliver Wendell Holmes. Medical Essays, 1842-1882, volume 9. Houghton, Mifflin, 1895.
  38. Symbols as a lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In AAAI Senior Member Track, 2021.
  39. Goal Recognition Design. In ICAPS, 2014.
  40. Privacy Preserving Plans in Partially Observable Environments. In IJCAI, 2016.
  41. Strong Stubborn Sets for Efficient Goal Recognition Design. In ICAPS, 2018.
  42. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668–2677. PMLR, 2018.
  43. Implicit communication in a joint action. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, pages 283–292. ACM, 2017.
  44. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215–289, 2018.
  45. The intelligent patient’s guide to the doctor-patient relationship: learning how to talk so your doctor will listen. Oxford University Press, 1998.
  46. Model-Based Contrastive Explanations for Explainable Planning. In XAIP Workshop, 2019.
  47. Explicable Robot Planning as Minimizing Distance from Expected Behavior. In AAMAS Extended Abstract, 2019a.
  48. A Unified Framework for Planning in Adversarial and Cooperative Environments. In AAAI, 2019b.
  49. Designing environments conducive to interpretable robot behavior. In IROS, 2020a.
  50. Signaling friends and head-faking enemies simultaneously: Balancing goal obfuscation and goal legibility. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’20, page 1889–1891. International Foundation for Autonomous Agents and Multiagent Systems, 2020b.
  51. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001.
  52. Exploring Computational User Models for Agent Policy Summarization. In IJCAI, 2019.
  53. Action Selection for Transparent Planning. In AAMAS, 2018.
  54. Deceptive Path Planning. In IJCAI, 2017.
  55. Tim Miller. Explanation in Artificial Intelligence: Insights from the Social Sciences. Artificial Intelligence, 2019.
  56. Planning over multi-agent epistemic states: A classical planning approach. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015.
  57. Planning for goal-oriented dialogue systems. arXiv preprint arXiv:1910.08137, 2019.
  58. Robust planning with incomplete domain models. Artificial Intelligence, 245:134–161, 2017.
  59. Generating diverse plans to handle unknown and partially known user preferences. Artificial Intelligence, 190(0):1 – 31, 2012.
  60. Nils J Nilsson. Principles of artificial intelligence. Morgan Kaufmann, 2014.
  61. Lies in the doctor-patient relationship. Primary care companion to the Journal of clinical psychiatry, 11(4):163, 2009.
  62. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  63. Artificial intelligence: a modern approach. Prentice Hall, 2002.
  64. Brian Scassellati. Theory of mind for a humanoid robot. Auton. Robots, 12(1):13–24, 2002.
  65. Making Hybrid Plans More Clear to Human Users – A Formal Approach for Generating Sound Explanations. In ICAPS, 2012.
  66. RADAR – A Proactive Decision Support System for Human-in-the-Loop Planning. In AAAI Fall Symposium, 2017a.
  67. Radar-a proactive decision support system for human-in-the-loop planning. In AAAI Fall Symposia, pages 269–276, 2017b.
  68. Ma-radar–a mixed-reality interface for collaborative decision making. ICAPS UISP, 2018.
  69. Not all users are the same: Providing personalized explanations for sequential decision making problems. arXiv preprint arXiv:2106.12207, 2021.
  70. Handling model uncertainty and multiplicity in explanations via model reconciliation. In Proceedings of the International Conference on Automated Planning and Scheduling, 2018a.
  71. Hierarchical Expertise Level Modeling for User Specific Contrastive Explanations. In IJCAI, 2018b.
  72. Model-free model reconciliation. In AAAI, 2019a.
  73. Why can’t you do that hal? explaining unsolvability of planning tasks. In International Joint Conference on Artificial Intelligence, 2019b.
  74. Planning with Explanatory Actions: A Joint Approach to Plan Explicability and Explanations in Human-Aware Planning. In AAAI, 2020a.
  75. –d3wa+–a case study of xaip in a model acquisition task for dialogue planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 488–497, 2020b.
  76. Bridging the gap: Providing post-hoc symbolic explanations for sequential decision-making problems with black box simulators. arXiv preprint arXiv:2002.01080, 2020c.
  77. Tldr: Policy summarization for factored ssp problems using temporal abstractions. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 30, pages 272–280, 2020d.
  78. Foundations of explanations as model reconciliation. Artificial Intelligence, 301:103558, 2021a.
  79. A unifying bayesian formulation of measures of interpretability in human-ai interaction. In International Joint Conference on Artificial Intelligence, pages 4602–4610, 2021b.
  80. Using state abstractions to compute personalized contrastive explanations for ai agent behavior. Artificial Intelligence, 301:103570, 2021c.
  81. Domain independent approaches for finding diverse plans. In IJCAI, pages 2016–2022, 2007.
  82. Generation of Policy-Level Explanations for Reinforcement Learning. In AAAI, 2019.
  83. Radar-x: An interactive interface pairing contrastive explanations with revised plan suggestions. In XAIP ICAPS, 2020.
  84. Kurt VanLehn. The behavior of tutoring systems. I. J. Artificial Intelligence in Education, 16(3):227–265, 2006.
  85. On exploiting hitting sets for model reconciliation. In AAAI, 2021.
  86. Manuela M Veloso. Learning by Analogical Reasoning in General Problem Solving. Doctoral Thesis, 1992.
  87. Contrastive Explanations for Reinforcement Learning in Terms of Expected Consequences. In IJCAI Workshop on explainable AI (XAI), 2018.
  88. H Wimmer and J Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception. Cognition, 1983.
  89. Graying the Black Box: Understanding DQNs. In ICML, 2016.
  90. Trust-aware planning: Modeling trust evolution in longitudinal human-robot interaction. arXiv preprint arXiv:2105.01220, 2021.
  91. Learning from ambiguous demonstrations with self-explanation guided reinforcement learning, 2021.
  92. A General Approach to Environment Design with One Agent. In IJCAI, 2009.
  93. Plan Explicability and Predictability for Robot Task Planning. In ICRA, 2017.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Sarath Sreedharan (41 papers)
  2. Anagha Kulkarni (13 papers)
  3. Subbarao Kambhampati (126 papers)