Probing Pretrained Language Models for Lexical Semantics (2010.05731v1)

Abstract: The success of large pretrained LLMs (LMs) such as BERT and RoBERTa has sparked interest in probing their representations, in order to unveil what types of knowledge they implicitly capture. While prior research focused on morphosyntactic, semantic, and world knowledge, it remains unclear to which extent LMs also derive lexical type-level knowledge from words in context. In this work, we present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks, addressing the following questions: 1) How do different lexical knowledge extraction strategies (monolingual versus multilingual source LM, out-of-context versus in-context encoding, inclusion of special tokens, and layer-wise averaging) impact performance? How consistent are the observed effects across tasks and languages? 2) Is lexical knowledge stored in few parameters, or is it scattered throughout the network? 3) How do these representations fare against traditional static word vectors in lexical tasks? 4) Does the lexical information emerging from independently trained monolingual LMs display latent similarities? Our main results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks. Moreover, we validate the claim that lower Transformer layers carry more type-level lexical knowledge, but also show that this knowledge is distributed across multiple layers.

Insights into Probing Pretrained LLMs for Lexical Semantics

The paper presented in "Probing Pretrained LLMs for Lexical Semantics" embarks on a comprehensive examination of pretrained LLMs (LMs) to identify and evaluate how these models represent lexical semantic knowledge. By focusing on a suite of empirical methods across six typologically diverse languages and five lexical tasks, the authors aim to uncover the nuanced ways in which LMs encapsulate word-level knowledge, questioning previous assumptions about their semantic capabilities.

Key Findings and Methodologies

The paper first addresses the diverse strategies for extracting lexical knowledge from LMs. Various configurations are explored, including monolingual versus multilingual LMs, in-context versus out-of-context word embeddings, and the inclusion of special tokens. These configurations impact the effectiveness of derived lexical representations on five major tasks: lexical semantic similarity (LSIM), word analogy (WA), bilingual lexicon induction (BLI), cross-lingual information retrieval (CLIR), and lexical relation prediction (RELP). The paper reveals certain general patterns such as:

• Best performance is often observed with monolingual LMs, emphasizing that language-specific training enhances lexical knowledge retention compared to multilingual counterparts like mBERT.
• Lexical knowledge extraction benefits from averaging over lower Transformer layers, aligning with the hypothesis that type-level semantic information tends to concentrate in these layers.
• Excluding special tokens like [CLS] and [SEP] from the lexical extraction process generally improves the quality of word representations.
• Contextual information derived from multiple embeddings of the same word in different sentences significantly enriches the extracted representations.

In addition to analyzing configurations, the paper compares state-of-the-art LMs with traditional static word embeddings (WEs) like fastText. The results demonstrate that BERT-derived embeddings often surpass traditional embeddings in monolingual tasks, though fastText remains competitive or superior in cross-lingual evaluations.

Theoretical and Practical Implications

The findings have substantial implications for the field of NLP, encouraging more targeted and task-specific approaches to NLP model utilization. They suggest a paradigm where researchers carefully select and fine-tune extraction strategies based on language and task-specific idiosyncrasies. This not only informs further research into lexical semantics encoding in neural architectures but also promises enhancements in practical applications ranging from machine translation to knowledge graph population.

Furthermore, these insights prompt intriguing questions about the future of static word embeddings, hinting at their potential redundancy with the advancement of large pretrained LMs. They suggest that while static embeddings have their uses, the dynamic and context-aware nature of embeddings derived from LMs offers a generally superior alternative for capturing the intricacies of lexical semantics.

Future Work

The paper outlines prospective research directions aimed at refining lexical extraction strategies and extending the scope to a broader set of languages and transformer architectures. It advocates for continued investigation into the optimal configurations for different tasks and highlights the need for more sophisticated methods to optimize layer averaging.

In conclusion, the systematic approach to evaluating how LMs encode lexical semantics provides substantial insight into their operation and utility. By dissecting various model configurations and assessing their performance across multiple tasks and languages, this work not only contributes to our understanding of LMs but also charts a path toward more effective and nuanced natural language processing.