Diachronic Word Embeddings

Updated 13 December 2025

Diachronic word embeddings are time-sensitive vector representations that quantify semantic evolution by mapping shifts in word context over distinct periods.
They employ methodologies such as static slice models, alignment strategies, and dynamic approaches to maintain temporal continuity and interpretability.
Applications include tracking cultural trends, monitoring social biases, and validating linguistic laws, making them essential for both historical and contemporary language studies.

Diachronic word embeddings are time-sensitive vector representations of lexical items designed to trace and quantify how word meanings and usage patterns evolve across distinct temporal intervals. By leveraging large temporally partitioned corpora and embedding algorithms, these models operationalize classic linguistic concepts of semantic drift, shift, and change, providing a powerful computational framework for investigating language dynamics, social change, cultural trends, and the emergence or disappearance of senses, connotations, and biases.

1. Theoretical Rationale and Types of Semantic Change

Diachronic word embeddings formalize the hypothesis that changes in the distributional context of a word reflect genuine changes in its meaning or social function. The principal types of semantic shift addressed are broadening, narrowing, amelioration, pejoration, substitution (technological shift), regular linguistic drift, and rapid cultural shift. Embeddings allow these phenomena to be operationalized as geometric displacements or neighborhood transformations in high-dimensional spaces. The field distinguishes between slow, regular sense drift (e.g. "meat" narrowing from "all food" to "animal flesh"), culturally induced shifts ("cloud" as 'object' to 'Internet storage'), and abrupt or policy-driven reorganizations, such as those documented during revolutionary periods in state-controlled media (Kutuzov et al., 2018, Ma et al., 12 Apr 2025).

2. Modeling Methodologies: Algorithms, Temporal Structuring, and Alignment

The dominant approaches to inducing diachronic embeddings can be classified as follows:

Static Slice Models: Independent training of distributional embeddings (e.g., PPMI+SVD, SGNS, CBOW, random indexing) for each time slice, followed by explicit alignment. Each model is optimized only on the corpus for its period, resulting in arbitrarily rotated spaces (Kutuzov et al., 2018, Tsakalidis et al., 2021, Dukić et al., 16 Jun 2025).
Alignment Strategies: Orthogonal Procrustes alignment (minimizing Frobenius norm between vector matrices across periods), local anchor-based alignment, and "compass" techniques (using a global context matrix). The canonical solution is:

$Q^* = \arg\min_{Q^\top Q=I}\|QX-Y\|_F$

with closed-form solution via SVD: $Y X^\top = U \Sigma V^\top$ , $Q = VU^\top$ (Dukić et al., 16 Jun 2025, Kutuzov et al., 2019).

Incremental and Dynamic Models: Incremental approaches warm-start later slices from earlier weights, implicitly aligning spaces and enhancing temporal continuity. Joint dynamic or regularized methods (e.g., Dynamic Bernoulli Embeddings, DSG) learn all time-specific representations simultaneously with explicit smoothness penalties (e.g., Gaussian random walks or $\ell_2$ penalties on $u_{w,t}-u_{w,t-1}$ ) (Yao et al., 2017, Montariol et al., 2019, Montariol et al., 2019, Gillani et al., 2019).
Contextualized and Generative Models: BERT-derived (contextual) embeddings are adapted for diachronic purposes via time-specific pooling of contextual representations (averaging/summing the last transformer layers for all occurrences per period), optionally after fine-tuning or domain pre-training on temporal slices (Qiu et al., 2022, Martinc et al., 2019). Recent generative methods (e.g., EDiSC) integrate word embeddings as priors within probabilistic sense-evolution models, crucial for low-resource or ancient-language settings (Zafar et al., 2023).

3. Metrics and Evaluation Tasks

Diachronic embedding models employ a range of quantitative metrics and tasks:

Cosine-distance measures: The prototypical semantic change metric is $1-\cos(u_{w,t},u_{w,t'})$ , quantifying displacement of a word across periods (Hamilton et al., 2016).
Neighborhood overlap / Jaccard / Kendall tau: Compare sets or rankings of k-nearest neighbors across slices.
Set-based metrics for relation tasks: In one-to-X analogical reasoning, standard information retrieval metrics (precision, recall, $F_1$ ) are used to evaluate relational predictions across time (Kutuzov et al., 2019).
Stability statistics: Two-way rotational mapping and stability scores account for distortions and directly quantify how stable a word's representation is after alignment (Guo et al., 2021).
Matrix and cluster approaches: Diachronic similarity matrices ( $S^w_{t,t'} = \text{cos}(e^{(t)}_w,e^{(t')}_w)$ ), followed by unsupervised clustering of matrix vectorizations, reveal recurring shift archetypes and permit the direct visualization of continuous and abrupt meaning change (Kiyama et al., 16 Jan 2025).
Bias/association measures: WEAT, MAC, and related metrics assess social bias and sentiment changes by comparing centroid distances between target groups and attribute sets across slices (Ma et al., 12 Apr 2025, Walter et al., 2021).
Mixed-effects modeling: Rates of semantic change are modeled statistically as a function of frequency, polysemy, and other lexical properties, uncovering universal laws (law of conformity and innovation) (Hamilton et al., 2016, Gupta et al., 2021).

4. Empirical Protocols: Datasets, Temporal Granularity, and Preprocessing

Corpus selection and partitioning strategies determine temporal resolution (yearly, decadal, event-based), directly impacting both statistical robustness and the detectability of semantic drift:

Corpus	Time Slices	Typical Token Volume
COHA, Google Books	Decades (1800–1990s)	$>20$ M per decade
Newswire/Gigaword	Yearly (1995–2017)	$>300$ M per year
Social Media (Twitter)	Months (April–June 2020)	$>10$ M per month
Croatian News	Five-year bins (2000–2024)	Up to $1.75$B per bin

Key preprocessing: segmentation, lemmatization, POS-tagging, multiword token merging, lowercasing, vocabulary thresholding, and removal of stopwords are critical for both robustness and interpretability. Corpus homogeneity and coverage (e.g., balancing for genre, avoiding domain drift) affect downstream alignment and evaluation (Dukić et al., 16 Jun 2025, Ma et al., 12 Apr 2025, Tsakalidis et al., 2021, Guo et al., 2021).

Diachronic word embeddings have enabled rigorous analysis across multiple domains:

Event prediction and digital humanities: Tracking emergence/termination of armed conflicts from news (location–insurgent relation modeling, improved $F_1$ via thresholding) (Kutuzov et al., 2019); tracing antisemitic/racist/ideological shifts in parliamentary, journalistic, and book corpora (Tripodi et al., 2019, Walter et al., 2021).
Cultural and linguistic comparison: Quantitative comparison of social stereotype evolution in Chinese versus Western state media, revealing both abrupt (policy-driven) and continuous (societal) cultural dynamics in encoded representations (Ma et al., 12 Apr 2025).
Lexical development and cognitive modeling: Charting the acquisition and adult-like convergence of semantic/syntactic categories in early child language, modeling frequency effects in stabilization (Gupta et al., 2021).
Quantitative linguistic laws: Empirically validating the law of conformity (rate of change $\propto [\textrm{frequency}]^{\beta_f}$ , with $\beta_f < 0$ ) and law of innovation (rate $\propto [\textrm{polysemy}]^{\beta_d}$ , with $\beta_d > 0$ ) across four languages and two centuries (Hamilton et al., 2016).
Sentiment and bias shift monitoring: Detecting contrastive sentiment trends in post-COVID newswire compared to health outcomes, and tracing gender or ethnic occupational bias in synchronic and diachronic perspective (Dukić et al., 16 Jun 2025, Gillani et al., 2019).
Modeling under resource constraints: Adapting regularized dynamic models and initialization schemes to low-resource or highly segmented settings, incorporating prior structure or shrinkage to distinguish true drift from noise (Montariol et al., 2019, Zafar et al., 2023).

6. Methodological Challenges, Limitations, and Open Directions

Alignment reliability: Orthogonal Procrustes and related techniques may be destabilized by severe vocabulary drift, sparse data, or drastic sense change. In such settings, global-anchor or second-order similarity methods can offer alternative metrics (Fomin et al., 2019).
Polysemy and sense-awareness: Most embedding trajectories reflect sense-averaged dynamics, which obscures sense-splitting or merging phenomena. Probabilistic models (EDiSC, DiSC, GASC) and contextualized embeddings with pooling or clustering have been proposed to target sense-specific dynamics, especially in low-resource or ancient-language corpora (Zafar et al., 2023, Qiu et al., 2022).
Temporal granularity, resource balance: The temporal resolution must trade off between signal detectability and data sufficiency. Child-language and social-media studies can afford monthly or sub-annual slices; century/corpus-wide analyses favor decadal bins (Gupta et al., 2021, Guo et al., 2021).
Data sparsity and regularization: For fine-grained studies and under multi-language, low-resource, or historical regimes, methods such as dynamic filtering, chronologically coupled regularization, and hard-thresholded drift penalization (HardShrink) are crucial to discriminate semantic shift from statistical artifact (Montariol et al., 2019, Montariol et al., 2019).
Evaluation standards: The field still lacks robust, high-quality gold-standard benchmarks for semantic shift, especially at scale and across languages. Where available, test sets derived from Oxford English Dictionary, manually annotated case studies (Russian shifts, child language), and historical event-relation datasets are used (Fomin et al., 2019, Gupta et al., 2021, Kutuzov et al., 2019).

Continuous-time and interpretable models: Emerging approaches treat embeddings as continuous, temporally parameterized functions, offering smoother alignment and fine-grained trajectory estimation (Yao et al., 2017).
Unsupervised trajectory clustering and visualization: Clustering word-level diachronic similarity matrices can expose common shift archetypes and enable scalable categorization of thousands of words’ semantic histories (Kiyama et al., 16 Jan 2025).
Integrating contextual embeddings: HistBERT and transformer-based methods demonstrate the gains from pre-training on balanced historical corpora and aggregating contextual embeddings for time-specific representations, though at significant computational cost relative to traditional approaches (Qiu et al., 2022, Martinc et al., 2019).
Cross-lingual, cross-genre, and cross-modal comparisons: Alignment and drift-regularized dynamic models have been successfully extended to paper parallel semantic change and convergent/divergent dynamics across languages and even genres (Montariol et al., 2019, Ma et al., 12 Apr 2025).
Bias, stereotype, and event analysis: Integration with social science frameworks (e.g., Bourdieu’s symbolic power) and modern bias/association tests (WEAT, MAC, ECT) highlights the use of diachronic embeddings in quantifying societal attitudes, historical bias propagation, and ideological shifts (Ma et al., 12 Apr 2025, Walter et al., 2021).

Diachronic word embeddings thus constitute a maturing but still rapidly evolving field, combining methodological rigor in distributional and probabilistic semantics with empirical depth across linguistics, social science, and computational applications. Continued advances in alignment, evaluation, data efficiency, and sense-awareness are critical for further progress.