Using Shapley interactions to understand how models use structure (2403.13106v2)

Abstract: Language is an intricately structured system, and a key goal of NLP interpretability is to provide methodological insights for understanding how LLMs represent this structure internally. In this paper, we use Shapley Taylor interaction indices (STII) in order to examine how language and speech models internally relate and structure their inputs. Pairwise Shapley interactions measure how much two inputs work together to influence model outputs beyond if we linearly added their independent influences, providing a view into how models encode structural interactions between inputs. We relate the interaction patterns in models to three underlying linguistic structures: syntactic structure, non-compositional semantics, and phonetic coarticulation. We find that autoregressive text models encode interactions that correlate with the syntactic proximity of inputs, and that both autoregressive and masked models encode nonlinear interactions in idiomatic phrases with non-compositional semantics. Our speech results show that inputs are more entangled for pairs where a neighboring consonant is likely to influence a vowel or approximant, showing that models encode the phonetic interaction needed for extracting discrete phonemic representations.

• The paper introduces Shapley Taylor Interaction Indices to quantify nonlinear feature interactions that uncover the underlying structure of diverse data modalities.
• It demonstrates distinct behavior across tasks by highlighting syntactic sensitivity in language models, phonetic patterns in speech, and edge-texture differences in image classification.
• The study offers a framework to enhance model interpretability and transparency, paving the way for multidisciplinary improvements in AI system design.

Shapley Interactions Shed Light on Model Behavior Across Diverse Data Modalities

Introduction

The intricacies of model interpretations in AI, particularly in understanding how models represent and interact with input features, have guided recent advancements toward more nuanced analytical methods. Among these, Shapley Taylor interaction indices (STII) have emerged as a pivotal tool in dissecting the non-linear relationships between features within models. By deploying STII, this investigation explores the underpinnings of data structure impacts on model representations across linguistic constructs, speech patterns, and visual cues.

Shapley Interactions and Their Computation

The roots of STII lie in game theory, specifically in the computation of Shapley values, which traditionally pertain to linear feature attributions. However, given the non-linearity inherent in deep learning models, Shapley values fall short of capturing complex feature interactions. To bridge this gap, STII employs a computation strategy that quantitatively assesses such interactions by considering the combined effect of feature sets on model outputs, opposed to isolating individual contributions.

• Shapley Taylor Interaction Indices (STII): Utilizes a discrete second-order derivative for approximating the interaction between pairs of feature sets within high-dimensional input spaces.
• Computation Challenge: The exhaustive calculation requisite for Shapley values, paired with the necessity to iterate over all possible feature subsets, renders the process computationally intensive, particularly for tasks with vast feature spaces like natural language processing or image classification.

Linguistic Structure Analysis

The paper's exploration into linguistic models unveils a nuanced understanding of how Transformer-based Masked LLMs (MLMs) and Autoregressive LLMs (ALMs) navigate syntax and idiomatic expressions. Through meticulous analysis, it surfaces a notable distinction between the two model types in handling syntactic distances and expressions:

1. Syntax and Distance: A correlation between syntactic proximity and STII was observed, with MLMs demonstrating a heightened sensitivity to syntactic structure in their interaction patterns, contrasting with ALMs' more distance-focused approach.
2. Idiomatic Expressions: Interaction intensities within idiomatic Multiword Expressions (MWEs) were discovered to be consistently stronger for MLMs, highlighting a divergence in how compositional semantics are tackled by different model architectures.

Speech Model Findings

Investigating speech models provided insights into the phonological underpinnings of speech recognition technologies. The paper identifies a direct relationship between phoneme articulation - the physical shaping of the oral cavity during speech - and the interaction of acoustic features. For example, the transition between consonants and vowels showcases a higher degree of non-linear interaction, aligning with phonetic principles that suggest vowels' acoustic features are heavily influenced by surrounding consonants.

Image Classification Insights

The exploration extends into the field of image classification, where pixel-based feature interactions offer a window into understanding object detection and boundary identification. The paper illustrates how pixels near object edges interact distinctively with those within object foregrounds versus backgrounds, providing empirical evidence for the conceptual divide between edge and texture in the visual perception domain.

Implications and Future Directions

The widespread applicability of STII across various modalities and tasks underscores its utility in uncovering the layered complexities of model representations. Furthermore, the paper propels forward the dialogue on integrating domain expertise into interpretability research, advocating for a multidisciplinary approach to uncovering new avenues for understanding AI systems.

• Theoretical Implications: Illuminating the nuanced relationship between model representations and underlying data structures, thereby enriching the theoretical foundations of AI interpretability.
• Practical Applications: Offering a blueprint for enhancing model transparency across sectors, enabling stakeholders to make informed decisions based on a deeper understanding of model behaviors.

Concluding Remarks

This comprehensive investigation into STII across linguistic, auditory, and visual tasks not only enriches the interpretability landscape but also champions a closer collaboration between AI research and domain-specific knowledge bases. As the quest for explainable AI advances, integrating these insights can lead to more transparent, trustworthy, and effective AI systems.