Interpretability Needs a New Paradigm (2405.05386v2)

Abstract: Interpretability is the study of explaining models in understandable terms to humans. At present, interpretability is divided into two paradigms: the intrinsic paradigm, which believes that only models designed to be explained can be explained, and the post-hoc paradigm, which believes that black-box models can be explained. At the core of this debate is how each paradigm ensures its explanations are faithful, i.e., true to the model's behavior. This is important, as false but convincing explanations lead to unsupported confidence in AI, which can be dangerous. This paper's position is that we should think about new paradigms while staying vigilant regarding faithfulness. First, by examining the history of paradigms in science, we see that paradigms are constantly evolving. Then, by examining the current paradigms, we can understand their underlying beliefs, the value they bring, and their limitations. Finally, this paper presents 3 emerging paradigms for interpretability. The first paradigm designs models such that faithfulness can be easily measured. Another optimizes models such that explanations become faithful. The last paradigm proposes to develop models that produce both a prediction and an explanation.

• The paper demonstrates that integrating faithfulness metrics into model design can improve the reliability and clarity of AI explanations.
• It critically examines intrinsic and post-hoc methods, emphasizing the risks of unfaithful yet plausible interpretations.
• By introducing approaches like FMMs and self-explaining models, the paper offers actionable insights for developing more transparent and accountable AI systems.

Exploring New Paradigms in Model Interpretability

Introduction to Interpretability Paradigms

Interpretability in ML refers to our ability to decipher, in simple human terms, why and how a model makes certain decisions. Traditionally, interpretability has been segmented into two dominant paradigms: the intrinsic and post-hoc approaches.

• Intrinsic paradigm: This viewpoint holds that models must be inherently interpretable; meaning clear, understandable decision processes must be woven into the architecture of the model itself. Classic examples include decision trees or linear models where the reasoning is straightforward and visible in the model's structure.
• Post-hoc paradigm: This perspective asserts that explanations can be derived from complex models (often considered "black-box" due to their opaque nature) after they have been trained. Techniques like feature importance derived from model outputs are used to interpret these models.

Both paradigms have their merits, but also significant limitations, leading researchers to propose and evaluate the emergence of new paradigms that might better address these flaws.

Limitations of Current Paradigms

The existing paradigms often fall short in terms of faithfulness, a term used to describe how accurately an explanation represents the operations and decisions of a model. Unfaithful explanations can be misleading, potentially causing more harm than good by engendering false confidence in the decisions made by AI systems.

• Intrinsic models: Though they provide a direct route to interpretability, they can be limited in performance and flexibility. Additionally, parts of even inherently interpretable models can remain opaque, such as certain layers in a neural network not directly contributing to the interpretability.
• Post-hoc explanations: These can be broadly applicable and useful, especially for complex models, but often at the cost of accuracy in the interpretation. They may fail to capture true causal relationships within model decisions, leading to potentially misleading interpretations.

Emerging Paradigms in Interpretability

Responding to the deficiencies in traditional paradigms, researchers have begun to sketch out potential new frameworks that can offer both high performance and faithful explanations:

1. Inherently Faithfulness Measurable Models (FMMs):
   • These models are designed not to be inherently interpretable directly but to make the measurement of an explanation's faithfulness straightforward and accurate.
   • A demonstrated approach involves modifying specific model types, such as RoBERTa, to accommodate direct and reliable faithfulness assessments without additional training or computational costs.
2. Models That Learn to Explain Faithfully:
   • Unlike traditional post-hoc methods, this paradigm focuses on optimizing models so that they naturally generate more faithful explanations.
   • This can involve novel training regimes or architectural tweaks that encourage the model to consider explanation quality during the training phase.
3. Self-explaining Models:
   • This concept pushes the idea further by suggesting that models should not only function well but also generate their own explanations as part of the output.
   • These models hold the potential for deep integration of interpretability, though ensuring the faithfulness of their self-generated explanations remains a critical challenge.

Future Directions and Caution

While these emerging paradigms show promise, they also introduce new complexities and risks. Ensuring the faithfulness of explanations remains paramount, as unfaithful but plausible explanations could lead to misguided trust in AI systems. Furthermore, the definition and measurement of faithfulness need to be precise and standardized to prevent inconsistencies and preserve the integrity of interpretability research.

Conclusion

The field of AI interpretability is at a crossroads, with significant opportunities for innovation in how we make complex models understandable and accountable. By exploring and developing new paradigms, we can hope to achieve models that are not only performant but also transparent and trustworthy in their decision-making processes. This exploration, while challenging, is crucial for the safe and ethical advancement of AI technologies.