On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs (2407.19200v2)

Abstract: Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. This surge in usage has led to an explosion in NLP model interpretability and analysis research, accompanied by numerous technical surveys. Yet, these surveys often overlook the needs and perspectives of explanation stakeholders. In this paper, we address three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their properties, and their relevance to different stakeholders. We further explore the practical implications of these paradigms by analyzing trends from the past decade across multiple research fields. To this end, we retrieved thousands of papers and employed an LLM to characterize them. Our analysis reveals significant disparities between NLP developers and non-developer users, as well as between research fields, underscoring the diverse needs of stakeholders. For example, explanations of internal model components are rarely used outside the NLP field. We hope this paper informs the future design, development, and application of methods that align with the objectives and requirements of various stakeholders.

• The paper presents a dual analysis that surveys over 14,000 studies and classifies key interpretability paradigms in NLP.
• It identifies a methodological shift from feature attributions to natural language explanations with clear disciplinary differences.
• The findings imply that harnessing LLMs for annotation and emphasizing user-friendly, causal methods can boost model transparency and stakeholder engagement.

Trends in NLP Model Interpretability in the Era of LLMs

In a comprehensive exploration titled "On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs", Nitay Calderon and Roi Reichart scrutinize the landscape of interpretability methods within the NLP domain. Their work epitomizes a dual approach: a survey of existing methodologies and an analytical piece that underscores the tendencies and divergences in interpretability research both inside and outside the NLP community.

Overview and Scope

The paper critically examines the surge in NLP model interpretability research, delineating how advancements in LLMs have catalyzed broader adoption and necessitated nuanced interpretability methods. By addressing three fundamental questions—why interpretability is needed, what aspects of models are interpreted, and how interpretations are achieved—the authors provide a granular analysis of interpretability paradigms and properties. They retrieve and analyze data from over 14,000 papers, leveraging an LLM to facilitate this process and surface insights that might inform future research trajectories.

Interpretability Paradigms and Properties

The authors classify interpretability methods into several paradigms based on what and how properties:

• Feature Attributions: Methods assigning relevance scores to input features, such as perturbations and gradients.
• Probing and Clustering: Probing involves classifiers predicting properties from model representations, whereas clustering interprets learned spaces through cluster analysis.
• Mechanistic Interpretability: Examines internal components of models, elucidating the functionality of neurons, layers, and circuits.
• Diagnostic Sets: Use specialized data subsets to assess model behavior on targeted properties.
• Counterfactuals and Adversarial Attacks: Generate counterfactuals or adversarial examples to understand model robustness and causal relationships.
• Natural Language Explanations (NLE): Extract or generate textual explanations for model predictions.
• Self-explaining Models: Models inherently designed for transparency, such as concept bottleneck models.

Each paradigm is scrutinized based on the explained mechanism (input-output, concept-output, input-internal, or internal-internal), scope (local or global), timing (post-hoc or intrinsic), accessibility (model-specific or model-agnostic), and form of presentation (scores, visualization, examples, texts).

Key Findings and Numerical Results

A systematic trend analysis reveals substantial differences between NLP developers (primarily responsible for internal model components) and non-developers (typically end-users or interdisciplinary researchers). The findings include:

• Stable Trends within NLP: Feature Attributions dominated early research but are witnessing a decline in favor of Natural Language Explanations, facilitated by improved text-generation capabilities in LLMs.
• Disciplinary Differences: While Feature Attributions and NLEs are prevalent across domains, certain methods like Mechanistic Interpretability and Adversarial Attacks are more common within NLP.
• Stakeholder Needs: Non-developers prefer methods such as LIME, SHAP, and clustering, which offer approachable tools outside NLP-specific contexts.

The dataset creation process utilized an LLM for accurate and scalable annotation, achieving over 92% accuracy against human annotation baselines, demonstrating a practical application of LLMs in metadata generation.

Implications and Future Perspectives

The implications of this research are manifold. For developers, the meticulous breakdown of what and how properties provides a framework for selecting and refining interpretability methods tailored to specific requirements. For policymakers and decision-makers, understanding stakeholder-specific needs emphasizes the significance of user-friendly, transparent, and faithful explanations.

One striking area of potential development lies in concept-level and causal-based methods which, despite their promising utility, remain underexplored. The authors advocate for leveraging LLM capabilities to advance research in these domains, ultimately achieving more accessible and faithful model explanations.

Conclusion

Calderon and Reichart's paper is an exhaustive resource that maps past research trajectories and future directions in NLP model interpretability. By dissecting methodologies and analyzing stakeholder needs, the authors shed light on a path forward that harmonizes technical rigor with practical, interdisciplinary applicability. The push for more user-centered and causally informed models marks a pivotal turn for NLP research, one that balances performance with transparency and accountability.