Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI (2409.16978v1)

Abstract: While Explainable AI (XAI) aims to make AI understandable and useful to humans, it has been criticised for relying too much on formalism and solutionism, focusing more on mathematical soundness than user needs. We propose an alternative to this bottom-up approach inspired by design thinking: the XAI research community should adopt a top-down, user-focused perspective to ensure user relevance. We illustrate this with a relatively young subfield of XAI, Training Data Attribution (TDA). With the surge in TDA research and growing competition, the field risks repeating the same patterns of solutionism. We conducted a needfinding study with a diverse group of AI practitioners to identify potential user needs related to TDA. Through interviews (N=10) and a systematic survey (N=31), we uncovered new TDA tasks that are currently largely overlooked. We invite the TDA and XAI communities to consider these novel tasks and improve the user relevance of their research outcomes.

Towards User-Focused Research in Training Data Attribution for Human-Centered Explainable AI

In this paper, Nguyen et al. propose a shift from a predominantly techno-centric approach in Explainable AI (XAI) research to a more user-centered methodology, specifically in the subfield of Training Data Attribution (TDA). Through a comprehensive two-stage needfinding paper, they explore the needs and preferences of AI practitioners, particularly model developers, to tailor TDA research towards practical and actionable insights. This essay provides a detailed analysis of their findings, methodology, and implications for the future of XAI and TDA.

Overview and Motivation

Explainable AI (XAI) aims to make AI systems transparent and interpretable to humans, facilitating trust and informed decision-making. However, XAI research has often been criticized for focusing excessively on theoretical and mathematical soundness, neglecting user-centric perspectives. This paper advocates for adopting a design-thinking-inspired, top-down approach to XAI research, ensuring that the developed tools align with the actual needs and practices of end-users.

TDA is an emerging subfield of XAI that links model behavior to specific training data, providing insights into the role of training samples in shaping model predictions. Despite its technical advancements, TDA research has been predominantly driven by solutionism and formalism, with limited consideration for user relevance. Nguyen et al. identify the need for refocusing TDA research on practical applications, especially in debugging AI models.

Methodology

The authors conduct a two-stage needfinding paper involving semi-structured interviews and a systematic survey to identify user needs related to TDA.

1. Interview Study: They recruit 10 participants with diverse backgrounds in high-risk AI application domains such as healthcare, law, and autonomous vehicles. These interviews aim to understand the participants' workflows, challenges, and perceptions of XAI and TDA. The thematic analysis reveals distinct user groups (system users vs. model developers) and emphasizes the centrality of data in model development and debugging.
2. Survey Study: Building on the interview findings, the authors design an interactive survey targeting model developers. Participants are presented with two model debugging scenarios (image classification and loan approval recommendation) and asked to customize their debugging interface based on different TDA attributes: action defining relevance (removal, label change, upweighting), metric (loss, ground truth probability, predicted class probability), and number of training samples (1-10).

Findings

1. Need for Flexibility and Reliability: The paper finds that TDA explanations should be adaptable to individual user needs and provide reliable, actionable insights. Users' preferences for TDA attributes are highly diverse, reflecting their hypotheses about model errors and previous experiences. For instance, some users prefer removal actions to understand causal effects, while others favor upweighting to focus on specific features.
2. Group Attribution: There is a clear preference for group attribution over individual sample attribution. Users find groups of relevant training samples more informative for understanding model errors and formulating effective debugging strategies.
3. Reliability of Explanations: Reliability is key for actionable TDA explanations. Users need confidence that the attribution scores will consistently reflect the impact of data-centric actions on overall model behavior.

Implications and Future Directions

The paper underscores the importance of grounding TDA and XAI research in actual user needs to enhance the practical relevance and effectiveness of these technologies. Several research directions are proposed:

1. Mental Models and TDA: Future research should explore how TDA influences and aligns with users' mental models, potentially leading to more intuitive and actionable explanations.
2. User-Centered Group Attribution: Developing methods to create meaningful groupings of training samples, considering factors like class labels and user intuitions, is crucial.
3. Holistic Understanding of Model Errors: Addressing broader issues beyond isolated errors, such as spurious correlations and dataset biases, can enhance the efficacy of TDA explanations.
4. Combating Solutionism: The paper highlights the need for interdisciplinary efforts to develop reliable and comprehensive TDA methods that consider the overall impact on model behavior.

Conclusion

Nguyen et al.'s paper marks a significant step towards user-centric XAI research. By shifting the focus from technical solutionism to practical applicability, they advocate for developing TDA methods that are flexible, reliable, and aligned with user needs. This approach is poised to enhance the usability and impact of AI systems across various high-stakes domains. The insights from their needfinding paper pave the way for future research focused on actionable and trustworthy TDA explanations, contributing to the broader goal of human-centered AI.