Using Captum to Explain Generative Language Models

Abstract: Captum is a comprehensive library for model explainability in PyTorch, offering a range of methods from the interpretability literature to enhance users' understanding of PyTorch models. In this paper, we introduce new features in Captum that are specifically designed to analyze the behavior of generative LLMs. We provide an overview of the available functionalities and example applications of their potential for understanding learned associations within generative LLMs.

• The paper introduces enhanced Captum methods combining perturbation and gradient-based attributions to elucidate generative language models.
• It details visualization techniques that map token contributions, uncovering model associations and potential biases.
• It demonstrates how few-shot prompt experiments affect sentiment predictions, highlighting the need for refined prompt engineering.

Explaining Generative LLMs Using Captum

The paper under discussion, "Using Captum to Explain Generative LLMs," elaborates on the integration of a set of new features in the Captum library tailored for enhancing the interpretability of Generative LLMs (GLMs). Vastly popular in natural language processing applications, GLMs such as GPT-3 demand clear explainability due to their complex key processes and increasing incorporation into critical areas like healthcare and law. As the need intensifies for transparency in model decision-making, Meta AI contributes profound advancements within the Captum library, specifically focusing on PyTorch models.

Overview of Captum's Attribution Methods

Primarily, the paper features various attribution methods available in Captum that are applicable to LLMs. Attribution methods, in this context, analyse model output sensitivity concerning input features. These are broadly categorized into perturbation-based and gradient-based methods, accentuating the flexibility of their application to GLMs.

The perturbation-based methods, such as Feature Ablation and Shapley Value Sampling, entail measuring the contribution of particular input features by observing the changes in outputs when features are altered or removed. For example, the Shapley Value Sampling provides a cooperative solution to distributing contribution scores amongst features efficiently, even though calculating exact Shapley Values is computationally excessive.

Conversely, gradient-based methods, including Saliency and Integrated Gradients, rely on the backpropagation of gradients through the network. Integrated Gradients is emphasized for its theoretical properties like sensitivity and implementation invariance, underscoring its robustness in attribute estimation.

LLM Attribution in Captum

The paper presents innovative functionalities in Captum v0.7, which focus on analyzing generative models. The examples showcase the Captum API utilities, allowing users to execute sophisticated experiments with minimal effort. For instance, users can evaluate how different tokens or segments in a prompt contribute to a model's output sequence, hence revealing underlying associations and behaviors.

Additionally, Captum's flexibility is extended through its visualization tools, facilitating the interpretation of attribution results. These visuals, such as heatmaps of feature importance, provide a tangible representation of how input features influence the model's predictions.

Applications in Understanding Model Behavior

Two practical applications of Captum's attributions are highlighted. The first addresses understanding model associations, revealing how particular features, such as location or occupation, may impact the model's generation of specific activities or characteristics. This is crucial for discerning any biases inherent in model predictions.

The second application pertains to evaluating the effectiveness of few-shot prompts. By measuring feature contributions in the context of sentiment analysis, the paper elucidates how prefatory few-shot examples can impact model outputs. Interestingly, it reveals that example prompts reduced confidence in positive sentiment prediction, indicating the need for further refinement in few-shot learning setups.

Implications and Future Directions

The augmented functionalities in Captum have significant implications for both practical tool development and theoretical studies in AI interpretability. The ability to customize baselines and features provides researchers ample flexibility in examining GLM behaviors, while visualization capabilities foster a deeper understanding of complex model dynamics.

The paper suggests prospective enhancements, such as automated baseline and feature selection, could streamline the interpretability process further. Moreover, millions utilizing generative models in production systems could benefit from improved tooling performance, an area earmarked for further research.

In essence, Captum's innovations symbolize a pivotal contribution to unraveling the complexities surrounding GLMs, equipping researchers with the tools required to illuminate the depth of LLM behaviors with precision.