Diachronic word embeddings and semantic shifts: a survey (1806.03537v2)

Abstract: Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications.

• The paper reviews current approaches for detecting semantic shifts by comparing word embeddings over time.
• It details methodologies like model alignment, global versus local measures, and incremental updates to capture dynamic language changes.
• The study highlights challenges such as evaluation benchmarks, non-English data limitations, and the need for standardized practices in diachronic analysis.

Overview of "Diachronic Word Embeddings and Semantic Shifts: A Survey"

The paper "Diachronic Word Embeddings and Semantic Shifts: A Survey" by Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal provides a structured examination of diachronic semantic analysis utilizing distributional methods, particularly prediction-based word embedding models. This work collates insights from natural language processing, computational linguistics, and other related fields, focusing on temporal changes in word meanings and their detection.

Semantic Shifts and Distributional Methods

Semantic shifts are the linguistic phenomena where words change meaning over time. Examples include cultural associations, such as geographical regions acquiring new connotations during conflicts. This paper highlights semantic shifts as a natural reflection of evolving language and societal changes. Using large corpora, researchers have progressively adopted computational approaches, particularly word embeddings, to capture these shifts more effectively.

The research covered in this paper is restricted to distributional word embedding models, which represent words as dense vectors across different time periods, allowing insight into changes in meaning. This survey emphasizes the need for common terminology and standardized practices in this emergent area of research.

Methodologies and Evaluation

Several methodologies are reviewed, highlighting the diversity of approaches in detecting semantic shifts. Key methodologies include:

1. Model Alignment: Aligning models from different time frames to enable meaningful comparison of word vectors across time. Techniques like orthogonal Procrustes transformations are often employed.
2. Global vs. Local Measures: Differentiating between methods using the entire vocabulary versus immediate neighbors. These approaches correspond to tracing linguistic and cultural shifts, respectively.
3. Incremental Updates: Training models sequentially across time frames, updating with new data to preserve historical semantic relationships.

The authors discuss various sources of diachronic data alongside challenges in evaluating such models, owing to limited gold-standard datasets for semantic shifts.

Applications and Future Directions

Diachronic embeddings have significant applications in understanding language evolution and real-world event detection. These methods can elucidate historical linguistic changes and provide insights into contemporary cultural trends, potentially influencing fields like digital humanities and socio-political analysis.

The survey acknowledges challenges including expanding research to non-English languages, improving methodologies for small datasets, and creating robust evaluation benchmarks. The development of formal mathematical frameworks and a deeper understanding of semantic shifts nature, such as differentiating types of shifts and their causes, remains a critical area for future research.

Conclusion

This paper highlights the significance of diachronic word embeddings in revealing lexical semantic transformations and underscores the complexities within this nascent field. It calls for greater cohesion and collaboration through specialized forums to address open challenges and push the boundaries of semantic shift detection and analysis.