Analysing Lexical Semantic Change with Contextualised Word Representations (2004.14118v1)

Abstract: This paper presents the first unsupervised approach to lexical semantic change that makes use of contextualised word representations. We propose a novel method that exploits the BERT neural LLM to obtain representations of word usages, clusters these representations into usage types, and measures change along time with three proposed metrics. We create a new evaluation dataset and show that the model representations and the detected semantic shifts are positively correlated with human judgements. Our extensive qualitative analysis demonstrates that our method captures a variety of synchronic and diachronic linguistic phenomena. We expect our work to inspire further research in this direction.

Analyzing Lexical Semantic Change with Contextualized Word Representations

The paper "Analyzing Lexical Semantic Change with Contextualized Word Representations" offers an innovative approach to lexical semantic change detection by leveraging the capabilities of state-of-the-art contextualized word representations, specifically those provided by BERT. This unsupervised method introduces a novel framework that enhances the accuracy and interpretability of semantic shift analysis over time.

Methodology Overview

The researchers employ BERT to generate contextualized usage representations for words within a given corpus, thereby capturing nuances in their meanings as influenced by surrounding word contexts. These representations are then clustered into usage types, which serves as a proxy for different senses or meanings of a word. Three main metrics—entropy difference, Jensen-Shannon divergence, and average pairwise distance—are proposed to quantify semantic changes across time intervals based on these usage type distributions.

Entropy Difference (ED) assesses shifts in lexical ambiguity or polysemy by measuring changes in uncertainty across usage type distributions. A significant increase in entropy suggests the emergence of new senses, while a decrease indicates narrowing of meanings.

Jensen-Shannon Divergence (JSD) provides a symmetric measure of divergence that captures subtle shifts in usage type frequencies, reflecting semantic evolution driven by social or cultural factors.

Average Pairwise Distance (APD) directly measures semantic drift using geometric distance between usage vectors, offering a robust alternative not dependent on predefined clustering.

Results and Implications

Correlation analysis against human judgments demonstrates the efficacy of the method in capturing meaningful semantic shifts. Notably, all three metrics show significant positive correlation with expert ratings in the provided evaluation dataset, underscoring the credibility of contextualized representations in semantic change modeling.

The researchers further exemplify the qualitative strengths of their approach by analyzing historical corpora. They highlight how technological advancements and socio-political phenomena have historically influenced language evolution—examples include narrowing interpretations of words like "coach" due to obsolescence of older meanings, and broadening senses observed in terms related to digital technology, such as "disk."

Potential Applications and Future Work

This paper paves the way for leveraging deep neural models to track linguistic phenomena over time, which has significant implications for fields such as historical linguistics, computational semantics, and digital humanities. It facilitates the automatic detection of novel usage patterns without extensive manual annotation, proving efficient for large-scale diachronic corpora.

Future research could explore methodologies for distinguishing types of semantic changes, such as metaphorization or simplification, further enriching linguistic theory. Additionally, extending this framework to other languages could uncover universal patterns of semantic evolution.

In summary, this paper advances the discourse on semantic change detection, demonstrating how contextualized embeddings from LLMs like BERT can provide fine-grained insights into the evolving lexicon, thereby enriching our understanding of language dynamics through an empirical lens.