TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations

Abstract: Machine Learning (ML) models are increasingly used to make critical decisions in real-world applications, yet they have become more complex, making them harder to understand. To this end, researchers have proposed several techniques to explain model predictions. However, practitioners struggle to use these explainability techniques because they often do not know which one to choose and how to interpret the results of the explanations. In this work, we address these challenges by introducing TalkToModel: an interactive dialogue system for explaining machine learning models through conversations. Specifically, TalkToModel comprises of three key components: 1) a natural language interface for engaging in conversations, making ML model explainability highly accessible, 2) a dialogue engine that adapts to any tabular model and dataset, interprets natural language, maps it to appropriate explanations, and generates text responses, and 3) an execution component that constructs the explanations. We carried out extensive quantitative and human subject evaluations of TalkToModel. Overall, we found the conversational system understands user inputs on novel datasets and models with high accuracy, demonstrating the system's capacity to generalize to new situations. In real-world evaluations with humans, 73% of healthcare workers (e.g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations. Our findings demonstrate that TalkToModel is more effective for model explainability than existing systems, introducing a new category of explainability tools for practitioners. Code & demo released here: https://github.com/dylan-slack/TalkToModel.

• The paper introduces TalkToModel, a system that uses adaptive dialogue and a conversational interface to explain complex ML predictions.
• The paper demonstrates its methodology by achieving over 70% parsing accuracy and high user approval in healthcare and ML professional studies.
• The paper highlights practical implications by democratizing ML explainability and outlining future research in deploying interactive, dialogue-based interfaces.

Explaining Machine Learning Models with Interactive Natural Language Conversations Using TalkToModel

This paper introduces TalkToModel, a novel system designed to improve the explainability of ML models through interactive natural language dialogues. The motivation stems from the increasing complexity of ML models used in industries like healthcare, finance, and law, where understanding model predictions is critical. Although traditional explainability tools exist, practitioners often struggle with their application due to the need to decide which methods to use and how to interpret their results properly.

System Architecture

TalkToModel is composed of three primary components:

1. Adaptive Dialogue Engine: This component processes natural language inputs, interpreting user queries into a structured dialogue that the system can understand.
2. Execution Engine: This part constructs explanations by selecting appropriate methods based on user queries, aiming to provide the most faithful and accurate results.
3. Conversational Interface: This interface allows users to interact with the system through open-ended dialogues, resembling human-like conversations.

These components together enable seamless interactions where users can probe into ML model predictions by merely talking to the system. TalkToModel supports a variety of conversational topics, such as discussing the importance of features, performing what-if analyses, and explaining model errors. For instance, healthcare practitioners can ask detailed questions like the impact of reducing a patient's blood glucose level on diabetes predictions.

Evaluation and Results

The effectiveness of TalkToModel was evaluated through empirical tests and human studies involving healthcare workers and ML professionals.

• Parsing Accuracy: For understanding user queries, TalkToModel employs pre-trained LLMs fine-tuned to parse user inputs to structured operations. The fine-tuned T5 models outperformed few-shot GPT-J in parsing accuracy, achieving above 70% overall accuracy. This indicates that the system can reliably understand user intents.
• User Studies: In terms of practical utility, 82.2% of healthcare workers found TalkToModel easier to use than traditional dashboards, while 84.6% of ML professionals concurred. Moreover, TalkToModel enabled users to complete tasks significantly faster, with higher accuracy and completion rates compared to conventional dashboards.

Implications and Future Work

The development of TalkToModel has several practical and theoretical implications:

• Practically, TalkToModel democratizes ML explainability by making it accessible to non-experts through intuitive dialogues. Its alignment with human communication patterns simplifies the understanding of complex models, particularly in high-stakes fields like medicine.
• Theoretically, the system provides a blueprint for leveraging natural language as an interface for dialogue systems in AI, hinting at future research avenues in integrating conversational AI with ML explainability.

Future developments could focus on deploying TalkToModel in real-world scenarios, such as clinics or financial institutions, to study its impact in situ. Additionally, enhancing the model's ability to generate responses grounded in detailed data and model operations could improve user trust and the system's utility.

Overall, TalkToModel presents a significant step towards bridging the gap between ML models' opacity and the transparency required by end-users, utilizing interactive, conversational approaches to explaining model behavior.