Open Sesame: Getting Inside BERT's Linguistic Knowledge (1906.01698v1)

Abstract: How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual representations like BERT perform well on tasks that require sensitivity to linguistic structure. We present here two studies which aim to provide a better understanding of the nature of BERT's representations. The first of these focuses on the identification of structurally-defined elements using diagnostic classifiers, while the second explores BERT's representation of subject-verb agreement and anaphor-antecedent dependencies through a quantitative assessment of self-attention vectors. In both cases, we find that BERT encodes positional information about word tokens well on its lower layers, but switches to a hierarchically-oriented encoding on higher layers. We conclude then that BERT's representations do indeed model linguistically relevant aspects of hierarchical structure, though they do not appear to show the sharp sensitivity to hierarchical structure that is found in human processing of reflexive anaphora.

Analysis of BERT's Encoding of Syntactic and Hierarchical Knowledge

The paper "Open Sesame: Getting Inside BERT's Linguistic Knowledge" conducts a detailed exploration of how BERT, a prominent transformer-based model, encodes linguistic information, with a specific focus on syntactically-sensitive hierarchical structure versus positionally-sensitive linear information. Two main investigative strategies are employed: diagnostic classification and diagnostic attention.

The paper begins with an investigation through diagnostic classification, a methodology probing BERT's embeddings for syntactic and linear cues. Using diagnostic classifiers, it examines BERT's capability of identifying hierarchically and linearly defined elements within sentences. Three distinct tasks are scrutinized: identifying the sentence's main auxiliary, subject noun, and the nth token. Key findings suggest that BERT's lower layers predominantly encode positional information, appropriate for the nth-token task. Positional information lessens as one moves to higher layers, with an observable enhancement in modeling hierarchical elements, as demonstrated by the main auxiliary and subject noun tasks. This reduction in linear cues coinciding with increased hierarchical processing hints at an intrinsic reorientation as the model progresses through layers. Notably, the performance altered with models of different scales, suggesting varying internal representations within the model's architecture depths.

The second approach evaluates BERT's attention mechanism, examining its adherence to linguistic structures in subject-verb agreement and reflexive anaphora contexts, among others. The introduction of a novel "confusion score" offers a nuanced metric for quantifying attention weights related to syntactic contexts. High confusion scores generally correlate with poor attention allocation, especially in the presence of distracting linguistic entities. Findings reveal BERT's moderate success in recognizing syntactic structures over simple, yet these structures often become obscure as distractors compound complexity or feature mismatches challenge the dependency resolution. Attention weights partly capture syntactic relationships but remain prone to misplaced emphasis on extraneous constituents. Intriguingly, attention progressively refines across layers, suggesting an iterative abstract refinement akin to human syntactical processing.

The implications of these findings are profound, both in practical deployments of BERT in NLP pipelines and theoretical understandings of how transformer-based models encapsulate linguistic nuances. Practically, these insights can inform enhancements in model interpretability, leading to more robust NLP applications capable of dealing with intricate syntactic dependencies. Theoretically, it positions BERT closer to replicating certain facets of human language processing, although discrepancies still demand further tuning and exploration.

Future research can pivot towards enhancing layer-specific diagnostics to visualize knowledge transitions across BERT's architecture. Understanding these transformations could unravel how linguistic robustness emerges and potentially inspire architectural innovations accommodating syntactic intricacies more intrinsically. Continued exploration of attention mechanisms and their role in syntactic encoding is pivotal to reaching closer approximations of human-like language understanding in artificial systems.