Probing for Bridging Inference in Transformer Language Models

Abstract: We probe pre-trained transformer LLMs for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower and middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider LLMs as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained LLMs substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to LLMs play an important role in the inference.